Method and system for postural analysis and measuring anatomical dimensions from a digital image using machine learning

ABSTRACT

A method for use of machine learning in computer-assisted anatomical prediction. The method includes identifying with a processor parameters in a plurality of training images to generate a training dataset, the training dataset having data linking the parameters to respective training images, training at least one machine learning algorithm based on the parameters in the training dataset and validating the trained machine learning algorithm, identifying with the processor digitized points on a plurality of anatomical landmarks in an image of a person displayed on a digital touch screen by determining linear anatomical dimensions of at least a portion of a body of the person in the displayed image using the validated machine learning algorithm and a scale factor for the displayed image, and making an anatomical circumferential prediction of the person based on the determined linear anatomical dimensions and a known morphological relationship.

TECHNICAL FIELD

This application relates to a method and system for using machinelearning algorithms in the measuring of anatomical dimensions from adigital image such as, for example, a two-dimensional orthree-dimensional image, on a mobile device or computer. Disclosedembodiments relate particularly to postural screening, measurements forclothing, estimating body composition such as body fat analysis, andmeasurements for orthotics or insert manufacturing.

BACKGROUND

Commonly-assigned U.S. Pat. Nos. 8,721,567, 9,801,550 and 9,788,759,each of which are hereby incorporated by reference in their entireties,provide improved methods and systems for measuring anatomical dimensionswhich overcome drawbacks and limitations in convention practice,including time consuming and imprecise postural deviation measurementsand the need for external equipment in the analysis or obtaining thepatient image, which can dictate that the screening be conducted in afacility having the required framework of vertical backdrop and plumbline or other equipment.

For example, U.S. Pat. No. 8,721,567 provides methods and systems forcalculating a postural displacement of the patient in a displayedcaptured image using a pixel to distance ratio for the displayed image.U.S. Pat. No. 9,801,550 provides methods and systems for acquiring aplurality of two-dimensional (2D) images of a person and making ananatomical circumferential measurement prediction of the person based onthe determined linear anatomical dimensions and a known morphologicalrelationship. U.S. Pat. No. 9,788,759 provides methods and systems foracquiring a plurality of three-dimensional (3D) images of a person andmaking an anatomical circumferential measurement prediction of theperson based on the determined linear anatomical dimensions and a knownmorphological relationship.

Despite these improvements in postural analysis and deriving anatomicalpredictions, substantial need still exists today in terms of developingsystems and practicing methods that are more efficient and accurate. Inthis regard, according to conventional practice, human interaction withdigital touch screen displays, cameras and image analysis can result inerrors and oversights. Moreover, conventional systems and methods docurrently employ any systematic learning procedures for making thesystems and methods “smarter.” Recently, the inventors have employedmachine learning techniques in developing systems and practicing methodsfor deriving an anatomical prediction that address these and otherdrawbacks.

Machine learning algorithms allow computer systems to solve a variety ofproblems, answer questions and perform other tasks based not solely uponpre-programmed instructions, but also upon inferences developed fromtraining data. The training data can be used to “train” the machinelearning algorithms by creating representations and generalizations thatcan then be applied to additional data in the future. The weights andparameters of the different representations and generalizations are“learned” by machine learning.

The inventors have looked to using machine learning algorithms andclassifier software implemented on a specialized computer to enhance,among other things, (1) digitizing points on a plurality of anatomicallandmarks on the displayed images, (2) determining linear anatomicaldimensions of at least a portion of a body of a person in the displayedimages using the digitized points and a scale factor for the displayedimages, and (3) making an anatomical circumferential measurementprediction of the person based on the determined linear anatomicaldimensions and a known morphological relationship. These algorithms andclassifiers utilize machine learning techniques to develop a model fordistinguishing and measuring various body dimensions. The training dataconsisting of image data may be annotated by a domain expert (such as aphysiologist) and fed into the classifier, and the classifier analyzesthe training data to identify patterns in the data that indicate when agiven sample corresponds to a known dimensions stored in a database.After the classifier has been trained, a set of similarly annotatedvalidation data is typically used to test the accuracy of theclassifier. This type of machine learning is known as “supervisedlearning,” since the training and validation data is annotated by ahuman “supervisor” or “teacher.” Unsupervised learning is alsocontemplated.

Use of machine learning algorithms has shown to provide faster, moreaccurate and more precise identification of anatomical landmarks thanpreviously possible. They do so, in part, based on the ability toidentify patterns in a myriad of data that are not discernable to thehuman eye or capable of being processed by any known process, mental orotherwise.

SUMMARY

In a first embodiment, there is provided a method for use of machinelearning in computer-assisted anatomical prediction. The methodcomprises identifying with a processor parameters in a plurality oftraining images to generate a training dataset, the training datasethaving data linking the parameters to respective training images,training at least one machine learning algorithm based on the parametersin the training dataset and validating the trained machine learningalgorithm, identifying with the processor digitized points on aplurality of anatomical landmarks in an image of a person displayed on adigital touch screen by determining linear anatomical dimensions of atleast a portion of a body of the person in the displayed image using thevalidated machine learning algorithm and a scale factor for thedisplayed image, and making an anatomical circumferential prediction ofthe person based on the determined linear anatomical dimensions and aknown morphological relationship.

In another embodiment, there is provided a system for use of machinelearning in computer-assisted anatomical prediction. The systemcomprises a memory configured to store at least one machine learningalgorithm and datasets, a processor programmed to: (i) identifyparameters in a plurality of training images to generate a trainingdataset, the training dataset having data linking the parameters torespective training images, (ii) train the machine learning algorithmbased on the parameters in the training dataset and validate the trainedmachine learning algorithm, (iii) identify digitized points on aplurality of anatomical landmarks in an image of a person displayed on adigital touch screen by determining linear anatomical dimensions of atleast a portion of a body of the person in the displayed image using thevalidated machine learning algorithm and a scale factor for thedisplayed image, and (iv) make an anatomical circumferential predictionof the person based on the determined linear anatomical dimensions and aknown morphological relationship.

In another embodiment, there is provided non-transitory computerreadable storage medium having stored therein a program to be executableby a processor for use of machine learning in computer-assistedanatomical prediction. The program causes the processor to executeidentifying with a processor parameters in a plurality of trainingimages to generate a training dataset, the training dataset having datalinking the parameters to respective training images, training at leastone machine learning algorithm based on the parameters in the trainingdataset and validating the trained machine learning algorithm,identifying with the processor digitized points on a plurality ofanatomical landmarks in an image of a person displayed on a digitaltouch screen by determining linear anatomical dimensions of at least aportion of a body of the person in the displayed image using thevalidated machine learning algorithm and a scale factor for thedisplayed image, and making an anatomical circumferential prediction ofthe person based on the determined linear anatomical dimensions and aknown morphological relationship.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a front perspective view of a mobile communication device withan image capturing device in the form of a camera on the back side,shown in dashed lines, of the device for acquiring an image of a patientand, as shown, a display screen on the front opposite side having atwo-dimensional array of pixels on which the image as seen on the camerais displayed.

FIG. 2 is a front perspective view of the screen of the device of FIG. 1showing a step of the postural screening method wherein a reference lineis overlaid the image providing vertical, horizontal and centerreferences on the display screen and wherein a corresponding referenceline is anchored to the displayed patient's image.

FIG. 3 is a front perspective view of the screen of the device of FIG. 1showing another step of the postural screening method wherein the tworeference lines in FIG. 2 have been aligned in the vertical or sagittalplane by rotation of the device relative to the patient being viewed bythe camera.

FIG. 4 is a front perspective view of the screen of the device of FIG. 1showing a further step of the postural screening method wherein the tworeference lines in FIG. 3 have been aligned in the vertical plane bytilting the device at the top toward the patient to level the imagecapturing device.

FIG. 5 is a front perspective view of the screen of the device of FIG. 1showing another step of the postural screening method wherein two spacedhorizontal lines are displayed on the screen at the top and bottom andthe image has been centered by panning and scaled by zooming with thecamera to fit the image precisely in the reference distance definedbetween the two lines to normalize the height of the image to a screendistance corresponding to a known number of pixels spanning the distancein the vertical direction.

FIG. 6 is a front perspective view of the screen of the device of FIG. 1showing an image of the patient like that of FIG. 5 but in the directionof the frontal plane of the patient.

FIG. 7 is a front perspective view of the screen of the device of FIG. 1wherein the image acquired in FIG. 5 optionally is displayed behind agrid overlay of vertical and horizontal lines against which aqualitative view of postural displacement can be observed.

FIG. 8 is a front perspective view of the screen of the device of FIG. 1wherein the image acquired in FIG. 6 optionally is displayed behind agrid overlay of vertical and horizontal lines against which aqualitative view of postural displacement is observed.

FIG. 9 is a process flow diagram of a method of postural screeningaccording to an example embodiment of the present invention.

FIG. 10 is a process flow diagram of acquiring an image of a patientwith the device of FIG. 1.

FIG. 11 is a front view of a subject for which an exemplary embodimentof the invention may make measurements either for postural screening asexplained with reference to FIGS. 1-10 or for measuring the dimensionsof the human body in the other embodiments as disclosed herein.

FIG. 12 is a front elevation view of an exemplary embodiment of theinvention, illustrating a front view of the subject depicted on thedigital touch screen display of the mobile device of the invention.

FIG. 13 is an alternate view of an exemplary embodiment of the inventionsimilar to FIG. 12, but illustrating a side view of the subject.

FIG. 14 is an alternate view of an exemplary embodiment of theinvention, illustrating a side view of the subject on the display of thedevice of the invention with the measurements being made.

FIG. 15 is a flow chart of operations possible with the mobile device ofan embodiment of the invention affording measurements for bodycomposition or clothing measurements.

FIG. 16 is a flow chart of steps for obtaining measurements forclothing, particularly tuxedo/suit fitting, in accordance with theinvention.

FIG. 17 illustrates and lists two dimensional linear measurements madewith the mobile device of the invention during the steps in theflow-chart of FIG. 16 and listing the clothing measurements calculatedfor fitting a tuxedo using the measurements and known mathematicalformulae.

FIG. 18 is a flow chart of steps for obtaining measurements forestimating body composition using the mobile device of the invention formaking measurements in accordance with the invention.

FIG. 19 is a front view and a side view of a subject depicting digitizedanatomical landmarks on the image and illustrating linear measurementsmade in the views of the subject in accordance with the steps of theflow chart of FIG. 18, the views being shown as part of a report on theresults of calculation of an estimate of average body fat using themeasurements.

FIGS. 20A-20D are three-dimensional analysis examples of postureanalysis (torso) using a mobile communication device with 3Dscanner/camera wherein the end user can pan and zoom and rotate toanalyze the subject in any plane;

FIG. 20A being a posterior oblique view depicting a left cervical listand right skull flexion relative to plane of the shoulders, and alsoshowing right shoulder lateral flexion.

FIG. 20B is a side view oblique where a anterior cervical list and athoracic posterior extension is noted.

FIG. 20C is a “bird's eye” view depicting right skull rotation relativeto plane of shoulders, and a right “rib hump” posture common inscoliosis.

FIG. 20D is a frontal view with slight oblique depicting and measuringleft head list with right skull flexion and right head rotation relativeto plane of shoulders, also showing subtle right shoulder flexion.

FIG. 21 illustrates a flowchart representation of a process ofgenerating a set of image-based training data and using that trainingdata to develop classifiers in accordance with an embodiment.

FIG. 22 illustrates a specialized computer or processing system that mayimplement machine learning algorithms according to disclosedembodiments.

FIG. 23 illustrates a machine learning algorithm according to anembodiment.

FIG. 24 illustrates a process executed by a machine learning algorithmaccording to an embodiment.

FIG. 25A, FIG. 25B and FIG. 25C illustrate a frontal image of a personprocessed by a machine learning algorithm according to an embodiment.

FIG. 26A, FIG. 26B and FIG. 26C illustrate a right-side image of aperson processed by a machine learning algorithm according to anembodiment.

FIG. 27A and FIG. 27B illustrate a posterior image of a person processedby a machine learning algorithm according to an embodiment.

FIG. 28A and FIG. 28B illustrate a left-side image of a person processedby a machine learning algorithm according to an embodiment.

DETAILED DESCRIPTION

According to embodiments, there are provided systems and methods ofusing machine learning algorithms to derive an anatomical predictionusing a known morphological relationship and a programmed apparatusincluding a digital touch screen display and a camera configured toacquire an image of a person on the digital touch screen display. Inembodiments, the systems and methods may comprise acquiring at least onedigital 2D or 3D image of a person on the digital touch screen display,digitizing points on a plurality of anatomical landmarks on thedisplayed three-dimensional image, calculating a circumferentialmeasurement of at least a portion of a body of a person in the displayedthree-dimensional image using at least the digitized points on thedisplayed three-dimensional image, and making an anatomical predictionbased on the calculated circumferential measurement and a knownmorphological relationship.

The disclosed system further includes means for making ananatomical-prediction using the measured dimensions and a knownmorphological relationship. Known mathematical formulae expressed in thecomputer program of the device relate the measured dimensions to theanatomical prediction. According to an aspect of the invention, theanatomical prediction includes at least one of circumference and volumeof a body part which may be displayed on the display screen. In onedisclosed embodiment the anatomical prediction is a clothing measurementselected from the group consisting of neck, overarm, chest, waist, hips,sleeve and outseam. According to another embodiment the anatomicalprediction is a body composition. In a further embodiment a posturaldisplacement is predicted from the measured dimensions and knownmorphological relationship. The disclosed system can also be used toobtain an image or images of the foot for use in orthotic and insertmanufacturing.

Thus, disclosed embodiments include a method of deriving an anatomicalprediction using a known morphological relationship and a programmedapparatus including a digital touch screen display and means foracquiring an image of a person on the digital touch screen display, themethod comprising acquiring an image of a person on the digital touchscreen display, digitizing points on a plurality of anatomical landmarkson the displayed image, determining linear anatomical dimensions of theperson's body using the digitized points and a scale factor for thedisplayed image, and making an anatomical prediction using thedetermined linear anatomical dimensions and a known morphologicalrelationship. In one embodiment the anatomical prediction is a clothingmeasurement. In another embodiment the anatomical prediction is bodycomposition. In a further embodiment the anatomical prediction includesat least one of circumference and volume of a body part. In a stillfurther embodiment the anatomical prediction is a postural displacementor is used for fitting/manufacturing an orthotic or insert for the foot.To further exemplify the use of the 3D acquired images in orthoticmanufacturing and clinical fitting, it is noted that in the past apractitioner such as a therapist or podiatrist typically must cast apatient's foot either weight bearing or non-weight bearing positions tohave the ability to capture the exact dimensions and form of the footfor orthotic manufacturing. Now instead of using a pin-type mold system,foam based or plaster type fitting system, in accordance with the methodand system of the invention a true 3D image can be acquired and a properorthotic manufactured therefrom saving considerable time and money tothe practitioner. Ultimately, the 3D data may also be used by a 3D type‘printer’ and an orthotic literally ‘printed’ based on 3D data.

For postural analysis in accordance with the present invention, the 3Dcaptured images have the advantage of traditional multi-camera systemsbut being much less expensive and mobile. Another advantage of 3Dpostural assessment in accordance with the invention is that the enduser practitioner can pan-zoom in any rotational view of the posture andthen click precise anatomical landmarks and planes which the system cancompare to one another to generate axial rotation (which is not easilycalculated from 2D photographs with a single camera).

For clothing fitting and body composition assessment, using a 3D imagein accordance with the disclosed method, exact circumferentialmeasurements can easily be derived with the 3D data from the Kinect orStructure type 3D camera/sensor. Clicking known locations permits one togenerate measurements of body parts and/or regions for exactanthropometric data from which can be extrapolated exact clothingfitting or precise body part measurements usually tracked in health andfitness. For example, personal trainers measure arms, legs and bodygirth of torso to track weight loss and “inches lost”. Further, usingexact circumferential data, a more precise method of anthropometric bodycomposition analysis is possible using known methods pioneered by theU.S. Department of Defense as discussed hereinafter.

In the disclosed embodiments, the method further comprises acquiring atleast two different views of the person as images on the digital touchscreen display and/or a digital 3D image of the person, which can berotated to provide each of said at least two different views, digitizingpoints on anatomical landmarks on each displayed image and determininglinear anatomical dimensions of the person's body using the digitizedpoints and a scale factor for each displayed image for making theanatomical prediction. In the disclosed embodiments the views acquiredinclude at least a front view and a side view of the person and/or adigital three-dimensional image of the person which can be rotated toprovide each of the front and side views.

An embodiment particularly relating to measuring dimensions for posturalscreening is disclosed but is understood as instructive with respect tothe other embodiments disclosed herein taken with the additionaldisclosure relating to each of the other embodiments.

The improved postural screening method according to the exampleembodiments of the present invention comprises acquiring an image of apatient on a display screen having an array of pixels, determining apixel to distance ratio for the displayed image, and calculating apostural displacement of the patient in the displayed image using thedetermined ratio. The standing framework of vertical backdrop and plumbline or overlaid grid-work of lines of the prior art are not necessary.According to the disclosed method, a known linear distance in thedisplayed image and the number of display screen pixels spanning thedistance are used in determining pixel to distance ratio. The knownlinear distance in an example embodiment is the height of the patient.Alternately, or in addition as a secondary calibration, a markeddistance can be provided in the acquired image of the patient, as by theuse of a meter stick in the image or other markings of a known distanceapart, to provide a known linear distance.

The postural screening method in example embodiments further includesscaling the size of the image relative to the display screen tonormalize the known linear distance in the image to a display screenreference distance corresponding to a known number of pixels fordetermining the pixel to distance ratio. According to a disclosedmethod, at least one reference line is provided over the displayed imageto demark the display screen reference distance.

The method as disclosed herein further includes displaying a referenceline overlaid on the screen providing vertical, horizontal and centerreferences, providing a corresponding reference line anchored to thedisplayed patient's image, and adjusting the image in the display sothat the two reference lines are aligned before determining the pixel todistance ratio.

In disclosed embodiments, the method further includes displaying areference line on the display screen over the acquired image, performingpanning to center the image on the screen, and performing zooming to fitthe image in the reference line before determining the pixel to distanceratio. Still further, the method comprises providing anatomicallandmarks on the acquired image of the patient to facilitate calculatinga postural displacement. The display screen is a touch screen for thispurpose to identify coordinates for calculation of posturaldisplacements by the programmed computer of the mobile, hand-heldcommunication device. An advantage of the 3D acquired images in themethod and system of the invention is that from these the operator caneasily predict axial rotations of different body regions compared with2D image acquisition which would typically require obtaining multiplephotographs from every side as well as axial (above the head or belowthe feet) to generate information needed to assess three dimensionalpostural analysis.

A system for performing postural screening according to the disclosedembodiments may comprise means for acquiring an image of a patient on adisplay screen having an array of pixels, means for determining a pixelto distance ratio for the displayed image and means for calculating apostural displacement of the patient in the displayed image using thedetermined ratio. The means for acquiring an image of a patientaccording to an example embodiment includes an image capture device ofthe mobile, programmed, hand-held communication device, for capturing atleast one of a 2D image or a 3D image of the person on the digital touchscreen display. Preferably, the device includes at least one positionaldevice selected from the group consisting of a gyroscope, anaccelerometer, and a level which provides a reference for leveling theimage capturing device. The system further includes means for panning adisplayed image on the screen to center the image on the screen, andmeans for zooming to fit a displayed image in a reference line on thedisplay screen. Means are provided for displaying at least one referenceline over the displayed image to demark a display screen referencedistance corresponding to a known number of pixels for determining thepixel to distance ratio. In the case of the use of a known 3D camera,the 3D system camera automatically calibrates using an infrared or othertype sensor so additional calibration of the image as described hereinwith respect to a 2D camera image may be unnecessary.

The system of disclosed embodiments further includes means fordisplaying a reference line overlaid on the screen providing vertical,horizontal and center references, means for displaying a correspondingreference line anchored to the displayed patient's image, and means foraligning image and display screen reference lines before determining thepixel to distance ratio. The system further includes means for providinganatomical landmarks on the acquired image of the patient to facilitatecalculating a postural displacement.

Disclosed embodiments further include a machine-readable mediumcontaining at least one sequence of instructions that, when executed,causes a machine to: calculate at least one postural displacement of apatient from a displayed image of the patient on a display screen havingan array of pixels, using a determined pixel to distance ratio for thedisplayed image.

Broadly, disclosed embodiments generally provide a method fordetermining an anatomical measurement of the human body such asmeasuring the dimensions of the human body comprising providing adigital anthropometer on a mobile device, and digitizing anatomicallandmarks on a displayed image such as a photograph or digitalthree-dimensional image of the human body displayed on the device withestablished calibration methods for measuring dimensions of the humanbody. And broadly, the embodiments of the prevention provide a digitalanthropometer device or system using digitization of anatomicallandmarks on a displayed image such as a photograph or digitalthree-dimensional image with established calibration methods. Thedevice/system is designed for measuring the dimensions of the human bodyand comprises a programmed device including a digital display, a touchscreen in the example embodiments having an array of pixels and a camerafor acquiring an image of a person on the digital touch screen display,and means for digitizing anatomical landmarks on an image of a persondisplayed on the touch screen display for measuring dimensions of thehuman body.

Another embodiment enables the ability to derive the anatomicalmeasurement such as the linear measurement or an angular measurementfrom the anterior, posterior and lateral aspects or 3D view of a bodypart, and then calculate an estimate of circumference and volume of thatbody part using mathematical equations.

Disclosed embodiments also enable recording a linear distance andsubsequent circumferential and volume calculations utilizingmathematical formulae which can also be tracked by software. Themeasurements can also be superimposed on the digital image of the persondisplayed on the device.

Another embodiment can produce reports for education on body posture,measurements for clothing, or for example body composition as explainedand shown with reference to FIGS. 11-19 below. This could be used byfitness professionals, health care professionals, or clothing industryprofessionals or where an anatomical measurements need to be calculatedby using prediction from digitizing anatomical points on/from a digitalpicture.

Once the images are obtained and digitized following protocols of thedisclosed methods, digitization points on anatomical landmarks forpurposes of posture, linear and circumferential anthropometricmeasurements can be performed. After these measurements are obtained,body ratios can be calculated to predict a person's body compositionusing well known anthropometric morphological relationships.

An exemplary embodiment may be utilized in health care, fitness or theclothing industry, to measure posture, to measure the foot formanufacturing orthotics and inserts, and to calculate body dimensions,shape, posture, and body composition based on anatomical ratiorelationship, and to track progress of linear, angular andcircumferential measurements. In other industries such as clothing, onecan obtain images, and find measurements needed to for example fit aperson for a suit or tuxedo instead of using manual tape measuring. Theimages used can be 2D or 3D images.

A first embodiment is a postural screening method comprising acquiringpatient information, acquiring an image of a patient, displaying areference line overlaid on the acquired image for scaling the acquiredimage, providing panning to center the acquired image, providing zoomingto fit the image within the displayed reference lines, for normalizingthe patient's height, determining a pixel to distance ratio using theacquired patient information and the normalized patient height,calculating postural displacements, and presenting a postural analysis.Aspects of the disclosed embodiments provide a postural screening methodthat may be implemented on a mobile, hand-held communication device thatincorporates the device's gyroscope, accelerometer, and camera. Thecamera may be either a 2D camera or a 3D camera such as a Kinect byMicrosoft, a Kinect type camera, a Structure Sensor by Occipital, or anysimilar technology.

Referring now to FIG. 1, a front perspective view of a mobile, hand-heldcommunication device 12 is shown, which on one side has a screen 13capable of displaying a frontal image 14 of a patient being viewed witha camera or image capture device on an opposite side. The device in theembodiment is an Apple iPhone 4 the computer of which is programmed inaccordance with the invention as described hereinafter to perform thedisclosed postural screening method. Other mobile, hand-heldcommunication devices capable of running a program in accordance withthe invention could also be used, such as iPhone®, iPod Touch®, iPad®and Android® devices including tablets and Windows® based tablets. FIGS.2-8 show front perspective views of screen 13 showing steps of a posturescreening method according to an embodiment of the present invention.Reference will be made to FIGS. 1-8 in the following description of thepostural screening method.

Referring now to FIG. 9, a postural screening method 50 is shownaccording to an embodiment of the present invention. Method 50 in theexample embodiment includes a step 52 of acquiring patient information,which may include, for example, accessing a database or prompting a userto enter information. Acquired information in may include, for example,height, weight, sex and age of a patient.

Method 50 may include a process 54 of acquiring an image of the patient.Referring now to FIG. 10, a process flow diagram of process 54 ofacquiring a frontal image 14 of the patient is shown. Process 54 asdisclosed includes a step 72 of activating an image capture device, inthis case the camera of the iPad 4. Process 54 in the embodimentincludes a step 74 of activating a positional device, namely one or moreof a gyroscope, an accelerometer, and a level in the device. Thepositional device(s) is used in accordance with the present invention toprovide feedback to a user as to whether the image capture device islevel.

Process 54 includes a step 76 of displaying a reference line overly 18on screen 13. The reference line overlay 18 may aid a user in aligningthe patient in the field of view of the image capture device byproviding, for example, a vertical reference 18 a, a horizontalreference 18 b, and a center reference 18 c. Process 54 includes a step78 if indicating a level patient. According to the embodiment of thepresent invention, in step 78 a visual indication including, forexample, corresponding references 16 a, 16 b, and 16 c, are providedanchored to frontal image 14. An aligned frontal image 14 may have areference line 20, which may have vertical, horizontal, and centerreference lines 20 a, 20 b, and 20 c, which may, for example, changecolors indicating alignment. Process 54 may also include a step 80 ofcapturing an image, for example, once alignment is achieved. In anexemplary embodiment of the present invention, a plurality of images maybe acquired including, for example, frontal image 14, lateral image 26,and a rear perspective image.

According to disclosed embodiments, process 54 may include accessing adata storage device. The data storage device may include, for example, apicture roll or album, which may contain a previously captured image ofthe patient. As another variation, the process 54 for acquiring an imageof a patient may include capturing a three-dimensional image of theperson by means of a 3D camera of the device 12 to display a digitalthree-dimensional image of the person on the digital touch screen. Thiswould involve taking several different views of the person as byscanning, for example. The user can pan and zoom and rotate thethree-dimensional displayed image to analyze the subject in any plane bymeans of computer input devices as discussed below.

Referring again to FIG. 9 method 50 may include a step 56 of displayingan upper reference line 24 a and a lower reference line 24 b over adisplay 22 of frontal image 14 and a lateral image 26 of the patient.The two spaced parallel lines are spaced apart a reference distancecorresponding to a known number of pixels of screen 13. The displayedreference lines 24 a and 24 b may be used as a reference for aligning ornormalizing the images 14 and 26, which may require positioning orscaling. Hence, method 50 may include a step 58 of providing panningcapability of the acquired image to a user, and a step 60 of providingzoom capability of the acquired image to a user. The provided panningcapability may allow a user to properly center or rotate images 14 and26 to fit in reference lines 24 a and 24 b. The provided zoom capabilitymay allow a user to properly size an acquired image to fit it withinreference lines 24 a and 24 b for normalizing the height of the patientin the acquired image and establishing a pixel height of the patient.Method 50 may include a step 62 of determining a pixel-to-distanceratio, which may be a quotient calculated by dividing a pixel height ofimages 14 and 26 divided by a patient's height.

Method 50 may include a step 64 of providing for identification of thepatient's anatomical landmarks, wherein a user may be prompted toidentify locations of a plurality of anatomical landmarks on theacquired image of the patient by touching the touchscreen of the deviceto identify an anatomical landmark. The plurality of the landmarks maycorrespond, for example, to skeletal landmarks, bone markings, orjoints. The identified plurality of landmarks may be used with the knownpixel to distance ratio for the displayed image to calculate absolutedistances and relative spatial positioning thereof, and may be used inan analysis of the patient's posture. In an exemplary embodiment of thepresent invention, the selection of anatomical landmarks may be on aplurality of images 14 and 26. The images of FIGS. 12-14 depict thedigitized anatomical landmarks placed on the image for the purpose ofmaking linear measurements in the front and side views of the subject.Where a 3D image is displayed, the image could be rotated and pan andzoomed to provide the front and side views as well as ‘bird's eye’ ofthe subject as well as other views as desired as seen in FIGS. 20A-20Ddescribed above.

Method 50 in the embodiment includes a step 66 of calculating posturaldisplacements using the determined pixel to distance ratio. Thedisplacements may include, for example, linear displacements and angulardisplacements. Method 50 may include a step 68 of presenting a posturalanalysis 27. Postural analysis 27 may display, for example, thecalculated linear or angular displacements 30, 34 and any deviationthereof from a normal or proper posture taking into account, forexample, the patient's age, sex, height, and weight. The normal orproper posture itself can be displayed over the displayed patient'simage to provide a visual comparison.

Specific elements of the disclosed systems and methods are described infurther detail below with respect to the acquisition, digitization,calculation and anatomical prediction features and the application ofmachine learning algorithms to these processes.

Acquisition & Digitization

The programmed device in one embodiment is a mobile, hand-heldcommunication device having at least one positional device selected fromthe group consisting of a gyroscope, an accelerometer, and a level tolevel the camera. With the device, the method for measuring includesactivating the at least one positional device and using an outputthereof for leveling the camera before capturing the image.

The display screen in the several embodiments is preferably a touchscreen for the purpose to quickly identify coordinates of the selectedanatomical landmarks of the body image displayed on the screen, e.g. todigitize the anatomical landmarks for calculation of linear distances bythe programmed computer of the device. These features advantageouslyreduce the time for measuring the dimensions and the accuracy, withoutthe need for external equipment or special facilities.

The patient's image can be acquired by accessing a database.Alternatively, the person performing the screening can operate an imagecapture device of a camera for acquiring the image of the patient.Either a 2D camera providing a 2D image or a 3D camera providing a 3Dimage can be used. The method preferably includes leveling the imagecapture device before capturing the image from which the pixel todistance ratio is to be determined for eliminating distortion. Accordingto the example embodiments, the image capture device and display screenare part of a mobile, hand-held communication device having at least onepositional device selected from the group consisting of a gyroscope, anaccelerometer, and a level. The method includes activating the at leastone positional device and using an output thereof to provide a referencefor leveling the image capturing device. Using the 3D type cameras suchas Kinect or Structure sensor by Occiptal, the 3D camera is tethered tothe mobile device directly or via wireless transmission, and providescalibrated 3D images on all three axes from which anatomical dimensionsand postural deviations can be derived.

Requirements of the mobile, hand-held communication device, thesoftware, and the interaction therebetween, and specific operations orsteps of the program for achieving the described functions of the methodfor an example embodiment are set forth below.

Leveling

Orientation Tracking

Requires an environment that can provide real-time or near real-timehorizontal and vertical orientation readings. These readings may beprovided by an “accelerometer”.

-   -   1. Begin reading the orientation data from the accelerometer.    -   2. Track each reading in a historical array of readings; do not        discard old readings.    -   3. When more than one reading has been tracked, apply a low-pass        filter against the newest and the historical readings. This will        provide accelerometer readings that more accurately reflect the        constant effects of gravity and reduce the influence of sudden        motion to the accelerometer.        Head-Up Display (HUD) Overlay

Requires a camera and a display screen that renders the camera's currentview. Requires an application programming interface that allows drawingand displaying images over the camera view on the display screen,partially obscuring portions of the camera view. Finally, requires apre-drawn graphic image files. The graphic image file may be partiallytransparent with one or more simple horizontal and vertical lines drawnon the image. The image file may also be more complex with circles,swirls, targets, multiple horizontal and vertical lines, etc. The imagefile will be used twice: once as stationary reference, once asdynamically moving indicator. While only one image is required thevisual design may be more appealing using two image files, one for eachusage.

-   -   1. Initialize the camera and viewpoint through normal methods of        those devices.    -   2. Using the programming interface and apply the image to the        display screen.    -   3. Using the programming interface, adjust the image location so        the image is viewable on the display screen. The camera display        screen should render both the camera's current view and the        image file. This image application will not be modified further        and serves the purpose of a stationary reference.    -   4. Using the programming interface and apply the image to the        display screen, again.    -   5. Using the programming interface, adjust the image location in        the exact same manner as the stationary image.    -   6. Using the programming interface, instruct the display to draw        the second image over the first stationary image.    -   7. The camera display screen should render the camera's current        view with both the image files drawn over the camera view,        partially obstructing the camera view.    -   8. The second image's location will be modified later and serves        the purpose of a movement indicator.        User Feedback-Leveling the Camera

Requires both the Orientation Tracking and the HUD Overlay methodsdescribed above. Orientation readings may be assigned x, y, and z planeswhich are discussed here as “roll”, “pitch”, and “yaw”.

-   -   1. Using the “roll” reading from the accelerometer, apply a        rotation to the movement indicator image of the HUD. The        programming interface of the display screen overlay will dictate        the angle units (i.e. radians, degrees) required to rotate the        movement indicator image. Use common angle mathematics to        convert the reading to radians or degrees as required.    -   2. Use the programming interface to apply a standard mathematic        rotation matrix to the movement indicator image's coordinate        system.    -   3. The movement indicator image should render partially rotated        on the camera display screen.    -   4. Using the programming interface or the operating system        documentation, determine the screen coordinates for the camera        display (for example, the iPhone 4S device boasts 960×640 pixel        display, however the iOS operating system assigns the size of        320×460; interest here is in the operating system size of        320×460; the operating system will handle conversion between the        device display ‘space’ and the operating system ‘space’).    -   5. Using the programming interface or the accelerometer        documentation, determine the minimum and maximum values of the        accelerometer “pitch” readings (for example, the iOS operating        system provides “pitch” readings as fractional decimal in the        range of −1.00 through +1.00).    -   6. Using the programming interface, read the current location        coordinate of the center of the movement indicator image.    -   7. Add or subtract the pitch reading to the vertical location        coordinate, restricting the value to the maximum and minimum        boundaries of the screen coordinates.    -   8. Using the programming interface, apply the result of the        addition (subtraction) to the movement indicator image.    -   9. The movement indicator image should be rendered on the camera        display screen in a different location. The image's center point        should remain within the viewable area of the display screen.    -   10. The software should continuously monitor the readings of the        accelerometer. With each new reading, update the rotation and        location coordinates of the movement indicator image as shown        above.    -   11. With one image stationary and a complimentary image moving,        the user will be able to visually notice when the images        perfectly overlap one another in both location and rotation.        This registration is their feedback that the device is oriented        correctly.        Display and Physical Measurements        Cropping

Requires a software environment that provides visual display elements(views) that can be nested inside of one another; allowing one elementto surround or envelope another. For example, the iOS operating systemprovides the UIView element (including UIView derivatives). Forreal-time cropping, requires a display screen that renders the views andany changes to the views (including size, scale, rotation, color,brightness, etc.).

-   -   1. Create two views, nested inside one another.    -   2. Load an image into the software (from a camera, disk drive,        computer memory, etc)    -   3. Using the programming interface to assign the image to the        inner view.    -   4. Optionally, use the programming interface to scale the inner        view to be larger than the outer view.    -   5. Optionally, use the programming interface to adjust the        location of the views so the inner view's boundaries extend past        the outer view equally in all directions.    -   6. Regardless of completing step 4 and 5, allow the user to        manipulate the inner view's size, scale, and location while        keeping the outer view fixed in both size, scale, and location.        Manipulation may occur by tracking the user input through any        computer input device. For example, on the iOS operating system        manipulation could be tracked by custom touch-screen readings or        standard pinch-and-zoom features.    -   7. After user manipulation has completed (indicated by an        arbitrary user action or input; for example pressing a “Done”        button) use the programming interface to read the current size        and position of both the inner and outer views.    -   8. Use the programming interface to capture the portion of the        inner view image that is currently within the outer view's        boundaries. Any portion of the inner view that extends past the        outer view's boundaries will be cropped and discarded.    -   9. The programming interface may require the cropping boundary        to be pre-calculated. The cropping boundary is used by the        programming interface and applied to the original image to        produce a new image from a portion of the original. The cropping        boundary can be calculated with simple arithmetic:        -   calculate (or read from the programming interface) the final            offset distance between the inner view and outer view's            center points,        -   calculate (or read from the programming interface) the final            resizing scale applied to the inner view,        -   use the offset divided by the scale to determine the origin            of the cropping boundary,        -   use the fixed size of the outer view divided by the scale to            determine the dimensions of the cropping boundary,        -   for example, the X coordinate of a cropping boundary            calculated in the iOS operating system would be:            x=outerview.contentOffset.x/outerview.zoomScale; and the            width of the cropping boundary would be:            width=outerview.frame.width/outerview.zoomScale.

As an example of calculating the cropping boundary, assume thefollowing:

-   -   An image of size 460×460    -   An outer view of size 300×400    -   The user has manipulated the inner image view to move it an        arbitrary direction and scaled to be twice as large. The result        of the manipulation is an image with effective size of 920×920        (×2 scale) with an offset of 195 in the X coordinate direction        and 289 in the Y coordinate.    -   The X coordinate of the cropping box would be 195/2=97.5 and the        width of the cropping box would be 300/2=150.    -   For reference, the Y coordinate in this example would be 144.5        and the height 200.    -   The programming interface should produce a new image from the        region of the original image with top left corner at 97.5,        144.5, width of 150 and height of 200.        Pixel Distance

Requires an image of an object cropped in a manner that the top andbottom of the object are at the edges of the top and bottom of theimage, and the physical height of the object must be known. Requires asoftware environment that can interpret image data and provide pixeldimensions of the image.

-   -   1. Load the image into the software (from a camera, disk drive,        computer memory, etc.)    -   2. Use the programming interface to read the pixel height of the        image    -   3. Divide the known height of the object by the pixel height        reading to determine the ratio of pixels to physical distance    -   4. The ratio can be used to calculate and convert any distance        of pixels to physical distances by multiplying the ratio and the        pixel distance

For example, given an image that is 1000 pixels in height and an objectthat is known to be 60 inches in height we can calculate:

-   -   Each pixel represents 0.06 physical inches: 60/1000=0.06    -   A distance of 250 pixels represents 15 physical inches:        0.06×250=15

Referring to FIG. 11, a subject 111 is illustrated whose measurementsmay be taken by an exemplary embodiment of the invention.

Referring to FIG. 12 and FIG. 13 an exemplary embodiment of theinvention is illustrated where the subject of FIG. 11 is displayed onthe display screen 13 of a mobile digital device 12. Anatomicallandmarks 116 digitized by the user's touching the screen at theanatomical landmarks thereon are highlighted on the side view of thesubject in FIG. 13. As it pertains to FIG. 12, the anatomical landmarks116 are illustrated on a front view of the subject.

Referring to FIG. 14, anatomical landmarks 116 are illustrated on thesubject in an exemplary embodiment of the invention. FIG. 14 alsoillustrates the measured distance from shoulder to elbow 118; themeasured distance from elbow to hand 200; the measured distance fromfront to back of chest 220 and the measured distance from front to backof waist 240. The flow charts of the steps for the methods are shown inFIGS. 15, 16 and 18. Images with digitized anatomical landmarks used inthe methods are shown on the displayed images in FIGS. 17 and 19.

Calculation & Anatomical Prediction

In the clothing measurement of FIGS. 16 and 17, the embodiment is aclothing fitting. In this case, the measurements are those needed for asuit or tuxedo. The measurements shown in the drawings are made orcalculated from linear measurements as shown. The circumferentialcalculations for neck, waist, hip and chest are made as described belowfor circumferential calculations from linear measurements. Additionally,the shirt sleeve length and outseam measurements are made as shown inFIG. 17.

In the body composition example of FIGS. 18 and 19, the applicationembodiment can be applied to measurements needed for body compositionanalysis which includes circumferential measurements (traditionallyperformed with a tape measure) for assessment of percentage body fat.Additionally one can calculate waist to hip ratio which is also acircumferential measurement. These are important health relateddiagnostic assessments with regards to body morphology and type.

Examples of known mathematical formulae useful in the severalembodiments include a body circumference formula employed in the exampleembodiments which utilizes measured body width (measured distance fromleft to right edges of body) and measured body depth (distance from backto front edges of body) made in front view and side view images of thebody, respectively.

The circumferential estimation is taken as the average of the results ofboth the equations (1) and (2) below. These are known formulas by amathematician and his formulae, referred to as the “Ramanujan'sformula”. The circumference of the ellipse with half axes a and b halfof the distance from each of the body width and body depth measurementsis given below where the approximation is from Ramanujan's formula:

$\begin{matrix}{{C \approx {\pi\left\lbrack {{3\left( {a + b} \right)} - \sqrt{\left( {{3a} + b} \right)\left( {a + {3b}} \right)}} \right\rbrack}} = {\pi\left\lbrack {{3\left( {a + b} \right)} - \sqrt{{10{ab}} + {3\left( {a^{2} + b^{2}} \right)}}} \right\rbrack}} & {{Equation}\mspace{14mu}(1)} \\{and} & \; \\{C \approx {{\pi\left( {a + b} \right)}{\left( {1 + \frac{3\left( \frac{a - b}{a + b} \right)^{2}}{10 + \sqrt{4 - {3\left( \frac{a - b}{a + b} \right)^{2}}}}} \right).}}} & {{Equation}\mspace{14mu}(2)}\end{matrix}$

If a=b then the ellipse is a circle with radius r=a=b and these formulasgive you C=2*pi*r.

Body composition in terms of body fat is calculated using the steps andmeasurements identified in FIGS. 17-19 then calculating circumferencefor neck, waist, abdomen and hip and obtaining the height and thenthrough data entry in one of the known formulae as set forth below whereall circumference and height measurements are in inches.% body fat=86.010×log 10(abdomen−neck)−70.041×log10(height)+36.76  Males.% body fat=163.205×log 10(waist+hip−neck)−97.684×log10(height)−78.387  Females.

Other known formulae describing known morphological relationships forbody fat could be employed as will be understood by the skilled artisan.For example, the results from several known formulae could be averaged.

Examples of known formulae are presented in the publications listedbelow, which are incorporated herein by reference:

-   Hodgdon, J. A. and M. B. Beckett (1984) Prediction of percent body    fat for U.S. Navy men from body circumferences and height. Report    no. 84-11, Naval Health Research Center, San Diego, Calif.;-   Hodgdon, J. A. Body (1990) Composition in the Military Services:    Standards & Methods. Report No. 90-21 Naval Health Research Center,    San Diego, Calif.    Application of Machine Learning Algorithms

Disclosed embodiments may further include machine learning algorithmsimplemented on specialized computers or computer systems for executingthe acquisition, digitization, calculation and anatomical predictionfunctions. In this regard, the algorithms may be used for automaticallyplacing points for postural analysis using commercial or open sourcetools; for example, face detection to determine the points for the eyes,or joint detection for measurement of limbs. Machine learning algorithmsmay be used for determining the outer boundaries of a body part in animage, assisting with the circumferential measuring of specific bodyareas (i.e., waist, neck, etc.), and mathematically processing a largedataset of known measurements, in order to create a regression formulathat will augment known measurement formulas or those disclosed herein.Machine learning algorithms may also be used in optimizing calculationsand increasing the precision and accuracy of predictive measurementalgorithms.

How effectively a machine learning algorithm can be trained may berelated to how well the data is classified or labeled before it is usedin a training procedure. Classifiers play an important role in theanalysis of 2D and 3D images and video of the human body. Inembodiments, classifiers are used to classify the body dimensions suchas, for example, body features, lengths, widths, etc., based on therelevant extracted body portions from the images. To develop a procedurefor identifying images or videos as belonging to particular classes orcategories (or for any classification or pattern recognition task),supervised learning technology may be based on decision trees, onlogical rules, or on other mathematical techniques such as lineardiscriminant methods (including perceptrons, support vector machines,and related variants), nearest neighbor methods, Bayesian inference,neural networks, etc.

Generally, classifiers require a training set consisting of labeleddata, i.e., representations of previously categorized media items (e.g.,body dimensions), to enable a computer to induce patterns that allow itto categorize hidden media items. Generally, there is also a test set,also consisting of labeled data that is used to evaluate whateverspecific categorization procedure is developed. In many cases, the testset is disjoint from the training set to compensate for the phenomenonof overfitting. In practice, it may be difficult to get large amounts oflabeled data of high quality. If the labeled data set is small, the onlyway to get any useful results at all may be to use all the availabledata in both the training set and the test set.

To apply standard approaches to supervised learning, the media segments(body dimensions) in both the training set and the test set must berepresented in terms of numbers derived from the images, i.e., features.The relationship between features extracted for the purposes ofsupervised learning and the content of an image or video has animportant impact on the success of the approach.

FIG. 21 illustrates a flowchart representation of a process ofgenerating a set of image-based training data and using that trainingdata to develop classifiers in accordance with an embodiment. As shownin the FIG. 21, the process begins with generation of clusters from animage's pixel data. Then, subsets of the original data set are createdcontaining only areas of interest. Finally, training and validation datasets are created, and the classifiers are trained and validated.

The programmatic tools used in developing the disclosed machine learningalgorithms are not particularly limited and may include, but are notlimited to, open source tools, rule engines such as Hadoop®, programminglanguages including SAS®, SQL, R and Python and various relationaldatabase architectures.

FIG. 22 illustrates a schematic of an example specialized computer orprocessing system that may implement machine learning algorithmsaccording to disclosed embodiments. The computer system is only oneexample of a suitable processing system and is not intended to suggestany limitation as to the scope of use or functionality of embodiments ofthe methodology described herein. The processing system shown may beoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with the processing system shown in FIG. 22 mayinclude, but are not limited to, personal computer systems, servercomputer systems, thin clients, thick clients, handheld or laptopdevices, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputersystems, mainframe computer systems, and distributed cloud computingenvironments that include any of the above systems or devices, and thelike.

The computer system may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.The computer system may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

The components of computer system may include, but are not limited to,one or more processors or processing units, a system memory, and a busthat couples various system components including system memory toprocessor. The processor may include a module that performs the methodsdescribed herein. The module may be programmed into the integratedcircuits of the processor, or loaded from memory, storage device, ornetwork or combinations thereof.

The computer system communicates with external devices such as a 2D or3D camera and may also communicate with one or more external devicessuch as a keyboard, a pointing device, a display, and the like, one ormore devices that enable a user to interact with computer system, and/orany devices (e.g., network card, modem, etc.) that enable computersystem to communicate with one or more other computing devices. Suchcommunication can occur via Input/Output (I/O) interfaces. The computersystem can communicate with one or more networks such as a local areanetwork (LAN), a general wide area network (WAN), and/or a publicnetwork (e.g., the Internet) via network adapter.

The above-described machine learning techniques are further described inthe following non-limiting example. By way of example, images frommultiple subjects may be stored in a database along with a multitude ofimage-related data including all relevant measurements, digitizedpoints, calculated parameters and predicted parameters. These data setsmay then be used to create training data sets that are tested andvalidated. Subsequent images processed may then be analyzed according tothe algorithms developed (i.e., the rules) and the digitized points maybe set according to the rules. The relevant calculations and predictionsmay then be based upon these digitized points and other hierarchal rulesknown to be particularly relevant to characteristic classifications ofthe image (e.g., weight, age, gender, body type, etc.). In turn, eachsubsequent image further trains the machine learning algorithms byvalidating, i.e., weighting, scoring, updating, etc. The result is amachine learning paradigm that automates and optimizes the anatomicalprediction process disclosed herein in ways not conventionally known.

In embodiments, anatomical landmark points may be automaticallyextracted using computer algorithms trained using supervised machinelearning techniques. Examples of common machine learning methods foranatomical landmark point detection include, but are not limited to,Active Shape Model (ASM), Active Appearance Model (AAM), Deformable PartModels and Artificial Neural Networks. In some embodiments, open sourcealgorithms such as OpenPose may be used for anatomical landmark pointsdetection. In all of these methods, a set of training images withannotated landmark points are used to build models. Once a model istrained and validated on a dataset, it is applied to detect landmarkpoints on novel images. In some practice, the training, validation, andtest image datasets are separated. Separate models may be trained andused to detect and extract anatomical landmark points in frontal, side,and other views. Some models may accommodate variations in camera viewangles.

The neural network may be a deep convolutional neural network. Theneural network may be a deep neural network that comprises an outputlayer and one or more hidden layers. In embodiments, training the neuralnetwork may include: training the output layer by minimizing a lossfunction given the optimal set of assignments, and training the hiddenlayers through a backpropagation algorithm. The deep neural network maybe a Convolutional Neural Network (CNN).

In a CNN-based model, a set of filters are used to extract features fromimages using convolution operation. Training of the CNN is done using atraining dataset containing images and landmark points, which determinesthe trained values of the parameters/weights of the neural network. FIG.23 depicts a CNN architecture for learning landmark points. As seen inFIG. 23, the CNN includes multiple layers. A convolutional layer mayinclude 8 128×128 kernels feeding into 2×2 pooling-layer. The poolinglayer then feeds into another convolutional layer including 24, 48×48kernels feeding into 2×2 pooling-layer. Further layers includefully-connected layers 1×256.

In some CNN models, the numbers of the CNN layers and fully connectedlayers may vary. In some network architectures, residual pass orfeedbacks may be used to avoid a conventional problem of gradientvanishing in training the network weights. The network may be builtusing any suitable computer language such as, for example, Python orC++. Deep learning toolboxes such as TensorFlow, Caffe, Keras, Torch,Theano, CoreML, and the like, may be used in implementing the network.These toolboxes are used for training the weights and parameters of thenetwork. In some embodiments, custom-made implementation of CNN and deeplearning algorithms on special computers with Graphical Processing Units(GPUs) are used for training, inference, or both. The inference isreferred to as the stage in which a trained model is used toinfer/predict the testing samples. The weights of a trained model arestored in a computer disk and then used for inference. Differentoptimizers such as the Adam optimization algorithm, and gradient descentmay be used for training the weights and parameters of the networks. Intraining the networks, hyperparameters may be tuned to achieve higherrecognition and detection accuracies. In the training phase, the networkmay be exposed to the training data through several epochs. An epoch isdefined as an entire dataset being passed only once both forward andbackward through the neural network.

An example application of the CNN model according to embodiments isillustrated in FIG. 24. As seen in FIG. 24, training images are markedwith annotated points of measurement.

The CNN model is trained based on the annotated points and weightingfactors, as described herein. Evaluation of the model using a validationset is conducted to validate the trained model. At this stage,hyperparameters may be tuned and the model retrained based on the tunedparameters. In any event, the best performing trained model isidentified and stored along with the relevant weighting factors. Oncethe training phase is complete, the process proceeds to the inferencephase where the trained and validated model is applied to a capturedimage of a body of a person and the resulting anatomical landmark pointsare plotted on the image.

FIGS. 25A-28B illustrate several images representative of this process.For example, FIGS. 25A, 26A, 27A and 28A illustrate images of a personin the frontal, right-side, posterior and left-side views, respectively.FIGS. 25B and 26B illustrate annotated training images of the frontaland right-side views, respectively. FIGS. 25C, 26C, 27B and 28Billustrate images of a person in the frontal, right-side, posterior andleft-side views, respectively, with the machine learned digitized pointsplotted on the respective images.

In some embodiments, a CNN-based network may be used for detection ofthe body silhouette. The detected body in original images is then usedin training or testing the network responsible for anatomical landmarkpoints' detection. In some networks, CNN-based methods such as You OnlyLook Once (YOLO) or DarkNet may be used for detection of the bodysilhouette. A bounding box may be used to show the position of adetected body in images. These two networks (body detection andanatomical landmark point detection) may be merged together. In thiscase, instead of training two separate but cascaded networks (a networkresponsible for human body detection in images and a network responsiblefor landmark point detection), one combined network is trained andutilized for both body detection and landmark extraction.

The network can be trained using a transfer learning mechanism. Intransfer learning, the network's weights are initially trained using adifferent image database than the posture image database to learn thedigitized points. Then, this pre-trained network is retrained furtherusing the images in posture database. The CNN architecture can be3-dimensional to handle 3D image data.

It should be understood, of course, that the foregoing relates toexemplary embodiments of the invention and that modifications may bemade without departing from the spirit and scope of the invention as setforth in the following claims. For example, clothing measurements arenot limited to tuxedo or suit measurements but could be made for otherclothing items, e.g. dresses, shirts, blouses, etc. Body composition isalso not limited to body fat but can include other estimations such asfor body mass index, waist-to-hip ratio, lean body mass, etc., usingknown morphological relationships. A three-dimensional image of thepatient's foot can also be used according to the disclosed method formaking measurements for making custom fit orthotics and inserts as willbe readily understood by the skilled artisan. Likewise, the anatomicalpredictions can include other predictions than those in the specificembodiments described herein without departing from the scope of theinvention as recited in the appended claims. Likewise, the digitaldisplay screen need not be a touch screen display as in the exampleembodiments but otherwise allowing, as by clicking a mouse, for example,to demark various anatomical landmarks thereon in accordance with theinvention, as will be readily understood by the person skilled in theart.

What is claimed is:
 1. A method for use of machine learning incomputer-assisted anatomical prediction, the method comprising:identifying with a processor parameters in a plurality of trainingimages to generate a training dataset, the training dataset having datalinking the parameters to respective training images; training at leastone machine learning algorithm based on the parameters in the trainingdataset and validating the trained machine learning algorithm;identifying with the processor digitized points on a plurality ofanatomical landmarks in an image of a person displayed on a digitaltouch screen by determining linear anatomical dimensions of at least aportion of a body of the person in the displayed image using thevalidated machine learning algorithm and a scale factor for thedisplayed image; and making an anatomical circumferential prediction ofthe person based on the determined linear anatomical dimensions and aknown morphological relationship.
 2. The method of claim 1, whereintraining the machine learning algorithm includes weighting theparameters in the training dataset and validating the machine learningalgorithm includes storing a best version of the machine learningalgorithm and the corresponding weighted parameters.
 3. The method ofclaim 1, wherein using the machine learning algorithm comprises:generating clusters from pixel data in the displayed image; creatingsubsets of the pixel data containing only areas of interest; identifyingat least one classifier in the subsets of the pixel data; generating atraining dataset based on the at least one classifier identified in thesubsets of the pixel data; and validating the at least one classifier inthe training dataset.
 4. The method of claim 1, wherein parameters inthe image of the person displayed on the digital touch screen are usedto further train and validate the machine learning algorithm by updatingcorresponding parameters in the training dataset.
 5. The method of claim1, wherein the parameters are at least one selected from the groupconsisting of weight, age, gender, body type, body silhouette, and bodymeasurement.
 6. The method of claim 5, wherein the body measurement is ameasurement selected from the group consisting of neck, overarm, chest,waist, hips, sleeve, and outseem.
 7. The method of claim 1, wherein themachine learning algorithm is at least one selected from the groupconsisting of an active shape model, an active appearance model, adeformable part model, and a neural network.
 8. The method of claim 1,wherein the machine learning algorithm is a deep convolutional neuralnetwork.
 9. The method of claim 8, wherein training the deepconvolutional neural network includes training an output layer byminimizing a loss function given an optimal set of assignments, andtraining hidden layers through a backpropagation algorithm.
 10. Themethod of claim 1, wherein the training step includes using a trainedmodel to infer testing samples by weighting the plurality of trainingimages relative to the parameters in the source dataset.
 11. The methodof claim 10, wherein the weighting includes optimizing using theplurality of training images relative to the parameters using an Adamoptimization algorithm or a gradient descent algorithm.
 12. The methodof claim 1, wherein the at least one machine learning algorithm includesa machine learning algorithm for identifying with the processor a bodysilhouette in the image of the person displayed on the digital touchscreen.
 13. The method of claim 1, wherein the machine learningalgorithm is trained using a transfer learning process comprising usingweights from corresponding parameters trained using a separate imagedataset to learn the digitized points.
 14. The method of claim 1,wherein the image of the person displayed on the digital touch screen isa 3D image.
 15. The method of claim 1, wherein the anatomicalcircumferential prediction is a clothing measurement, a bodymeasurement, or a postural measurement.
 16. The method of claim 1,wherein the scale factor is a ratio of pixel to distance.
 17. The methodof claim 1, further comprising plotting the plurality of anatomicallandmarks on the image of the person displayed on the digital touchscreen.
 18. The method of claim 1, further comprising displaying theanatomical circumferential prediction on the image of the persondisplayed on the digital touch screen.
 19. A system for use of machinelearning in computer-assisted anatomical prediction, the systemcomprising: a memory configured to store at least one machine learningalgorithm and datasets; a processor programmed to: (i) identifyparameters in a plurality of training images to generate a trainingdataset, the training dataset having data linking the parameters torespective training images; (ii) train the machine learning algorithmbased on the parameters in the training dataset and validate the trainedmachine learning algorithm; (iii) identify digitized points on aplurality of anatomical landmarks in an image of a person displayed on adigital touch screen by determining linear anatomical dimensions of atleast a portion of a body of the person in the displayed image using thevalidated machine learning algorithm and a scale factor for thedisplayed image; and (iv) make an anatomical circumferential predictionof the person based on the determined linear anatomical dimensions and aknown morphological relationship.
 20. A non-transitory computer readablestorage medium having stored therein a program to be executable by aprocessor for use of machine learning in computer-assisted anatomicalprediction, the program causing the processor to execute: identifyingwith a processor parameters in a plurality of training images togenerate a training dataset, the training dataset having data linkingthe parameters to respective training images; training at least onemachine learning algorithm based on the parameters in the trainingdataset and validating the trained machine learning algorithm;identifying with the processor digitized points on a plurality ofanatomical landmarks in an image of a person displayed on a digitaltouch screen by determining linear anatomical dimensions of at least aportion of a body of the person in the displayed image using thevalidated machine learning algorithm and a scale factor for thedisplayed image; and making an anatomical circumferential prediction ofthe person based on the determined linear anatomical dimensions and aknown morphological relationship.